Start target metrics server on target startup #1097
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…than experiment started signal. Signed-off-by: Caleb Metz <[email protected]>
cmetz100
force-pushed
the
cmetz/target_metrics_start_on_target_startup
branch
from
3652511
to
4afad48
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Changes the target metrics server to wait on target running signal rather than experiment started signal.
Motivation
Currently we start the capture manager on target startup which means we are technically able to start both the observer and target metrics server on startup as well. We currently do this for the observer which gives us insight as to the targets startup behavior from the perspective of lading metrics (currently useful for debugging mainly). However it may be useful to also gain some insight as to the targets internal view of its behavior before we start the experiment (during warmup period).
Note: I expect the first few polls to fail as the prometheus end points may not be exposed as soon as we signal agent as running
Related issues
Additional Notes
I am not 100% confident we actually want this in main, open to peoples thoughts.
For context I want this for investigating the agents startup behavior in dogstatsd experiments. Planning on using this version of lading in some test jobs. This pairs with my pr in smp to reallow the exporting of warmup metrics.